Predicting Body Movement and Recognizing Actions: an Integrated Framework for Mutual Benefits

Human action recognition and body movement prediction are important tasks. They are different and have traditionally been addressed separately. These tasks, however, provide mutual benefits to each other, and existing methods fail to capture these benefits. In this paper, we propose a method for jointly recognizing the action and predicting the movement of a person. Our method is based on two Long-Short Term Memory (LSTM) recurrent neural networks, but extend them to provide and receive benefits of each other. In particular, we design two LSTM architectures. One LSTM can generate a sequence of body movement conditioned on the past movement and the predicted class of the action, and the other LSTM can recognize the human action based on the predicted sequence of body movement. Experiments on Montalbano and MSR Action 3D datasets show that movement prediction provides benefits to early recognition of human action, which in turn improves the quality of the predicted movement.